Image correction across multiple spectral regimes

ABSTRACT

Systems, including apparatus and methods, for obtaining and/or correcting images, particularly from atmospheric and/or other distortions. These corrections may involve, among others, determining corrective information in a first (e.g., visible) wavelength regime, and then applying the corrective information in a second (e.g., longer) wavelength regime, such as infrared (IR) or millimeter-wave (MMW) wavelengths, in real time or with post-processing. For example, these corrections may include scaling a phase diversity correction from one wavelength to another. These systems may be useful in any suitable imaging context, including navigation, targeting, search and rescue, law enforcement, and/or surveillance, among others.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/207,536, filed Aug. 19, 2005, now U.S. Pat. No. 7,515,767, which inturn is based upon and claims the benefit under 35 U.S.C. § 119(e) ofU.S. Provisional Patent Application Ser. No. 60/696,299, filed Jul. 1,2005. These applications are incorporated herein by reference in theirentireties for all purposes.

INTRODUCTION

Optical systems may be used to form and/or record images of objectsand/or scenes. Unfortunately, when an optical system obtains imagesbased on image data that have passed through a medium, the imagesobtained generally will be distorted both by the medium, and by thecomponents of the optical system itself. For example, the image of anobject viewed with a telescope or other long-range imaging system may bedistorted both by atmospheric effects (e.g., the scintillation,convection, turbulence, scatter, and varying index of refraction of theatmosphere, among others, which can induce various spatial and temporalperturbations in the incoming wavefront, etc.), and by mechanical,thermal, and optical limitations of the instrument (e.g., path-lengtherror introduced by out-of-focus components of the field of view,limitations on the collection of spatial frequencies imposed by theobjective aperture, uncorrected aberration in the objective lens, mirrordeformations generated by supporting devices, etc.). These distortionsoccur, for example, when ground-based telescopes (or other imaginginstruments) obtain images of objects on the ground, in the air, or inspace, and when airborne or space-based telescopes (or other imaginginstruments) in aircraft or on satellites obtain images of objectswithin Earth's atmosphere, such as objects on or near Earth's surface.This also may occur in situations in which an imaging system and theobject to be imaged are separated primarily horizontally, or bothhorizontally and vertically, by a portion of the Earth's atmosphere.

The effects of atmospheric distortion can significantly limit imageresolution. For example, atmospheric distortion can limit the best“seeing conditions” to approximately 1 microradian at high-altitudeastronomical observatories, looking straight up. The limiting resolutionbecomes rapidly worse for lower-altitude and near-horizontal viewingscenarios typical for cameras and electro-optical systems.

Various methods have been developed to mitigate or eliminate the effectsof image distortion. These methods generally rely on obtainingcorrective information within the wavelength regime(s) in which imagingdata is desired. For example, visible image data are used to correctvisible images, and infrared image data are used to correct infraredimages. However, this may be prohibitively expensive—or otherwiseimpractical—as a technique to correct relatively long-wavelength images,statically or in real time, due in part to the difficulty and expense ofrapidly collecting and processing image data in such regimes, includingthe additional hardware complexity needed for the infrared. This isespecially true with phase diversity techniques, which may use multipleimages to obtain the needed image correction information. In such cases,a need exists for an effective and practical means of eliminating, or atleast mitigating, atmospheric distortion effects.

SUMMARY

The present teachings provide systems, including apparatus and methods,for obtaining and/or correcting images, particularly from atmosphericand/or other distortions. These corrections may involve, among others,determining corrective information in a first (e.g., visible) wavelengthregime, and then applying the corrective information in a second (e.g.,longer) wavelength regime, such as infrared (IR) or millimeter-wave(MMW) wavelengths, in real time or with post-processing. For example,these corrections may include scaling a phase diversity correction fromone wavelength to another. These systems may be useful in any suitableimaging context, including navigation, targeting, search and rescue, lawenforcement, and/or surveillance, among others.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic diagram showing visible light from an object beingsplit into two phase diverse beams in preparation for correcting avisible image using the phase diversity method of blind deconvolution.

FIG. 2 is a schematic diagram showing how measured phase diversity datamay be used to correct imagery in various wavelength regimes.

FIG. 3 is a schematic diagram showing image data being split into oneprimarily infrared beam and two primarily visible phase diverse beams,with the resulting corrective information being processed and used tocorrect an image formed from the infrared beam, using the phasediversity method of blind deconvolution.

FIG. 4 shows a representative aircraft equipped with an image correctionsystem, according to aspects of the present teachings.

DEFINITIONS

Technical terms used in this disclosure have the meanings that arecommonly recognized by those skilled in the art. However, the followingterms may have additional meanings, as described below. The wavelengthranges identified in these meanings are exemplary, not limiting, and mayoverlap slightly, depending on source or context. The wavelength rangeslying between about 1 nm and about 1 mm, which include ultraviolet,visible, and infrared radiation, and which are bracketed by x-rayradiation and microwave radiation, may collectively be termed opticalradiation.

Ultraviolet radiation. Invisible electromagnetic radiation havingwavelengths from about 100 nm, just longer than x-ray radiation, toabout 400 nm, just shorter than violet light in the visible spectrum.Ultraviolet radiation includes (A) UV-C (from about 100 nm to about 280or 290 nm), (B) UV-B (from about 280 or 290 nm to about 315 or 320 nm),and (C) UV-A (from about 315 or 320 nm to about 400 nm).

Visible light. Visible electromagnetic radiation having wavelengths fromabout 360 or 400 nanometers, just longer than ultraviolet radiation, toabout 760 or 800 nanometers, just shorter than infrared radiation.Visible light may be imaged and detected by the human eye and includesviolet (about 390-425 nm), indigo (about 425-445 nm), blue (about445-500 nm), green (about 500-575 nm), yellow (about 575-585 nm), orange(about 585-620 nm), and red (about 620-740 nm) light, among others.

Infrared (IR) radiation. Invisible electromagnetic radiation havingwavelengths from about 700 nanometers, just longer than red light in thevisible spectrum, to about 1 millimeter, just shorter than microwaveradiation. Infrared radiation includes (A) IR-A (from about 700 nm toabout 1,400 nm), (B) IR-B (from about 1,400 nm to about 3,000 nm), and(C) IR-C (from about 3,000 nm to about 1 mm). IR radiation, particularlyIR-C, may be caused or produced by heat and may be emitted by an objectin proportion to its temperature and emissivity. Portions of theinfrared having wavelengths between about 3,000 and 5,000 nm (i.e., 3and 5 μm) and between about 7,000 or 8,000 and 14,000 nm (i.e., 7 or 8and 14 μm) may be especially useful in thermal imaging, because theycorrespond to minima in atmospheric absorption and thus are more easilydetected (particularly at a distance). The particular interest inrelatively shorter wavelength IR has led to the followingclassifications: (A) near infrared (NIR) (from about 780 nm to about1,000 nm), (B) short-wave infrared (SWIR) (from about 1,000 nm to about3,000 nm), (C) mid-wave infrared (MWIR) (from about 3,000 nm to about6,000 nm), (D) long-wave infrared (LWIR) (from about 6,000 nm to about15,000 nm), and (E) very long-wave infrared (VLWIR) (from about 15,000nm to about 1 mm). Portions of the infrared, particularly portions inthe far or thermal IR having wavelengths between about 0.1 and 1 mm, mayalternatively, or in addition, be termed millimeter-wave (MMV)wavelengths.

DETAILED DESCRIPTION

The present teachings relate to systems, including apparatus andmethods, for obtaining images and/or correcting images, particularlyfrom atmospheric and/or other wavefront errors and distortions.Obtaining images, as used herein, may include optically forming aduplicate, counterpart, and/or other representative reproduction of anobject or scene, especially using a mirror (reflective optic) and/orlens (refractive optic). The duplicate, counterpart, and/orreproduction, in turn, may be detected, in analog or digital formats,especially using analog (e.g., film) and/or digital (e.g., focal planearrays) recording mechanisms. Correcting images, as used herein, mayinclude determining corrective information at a first wavelength, orrange of wavelengths, and then applying the corrective information to animage at a second wavelength, or range of wavelengths, in real time orwith post-processing. The first wavelength, or range of wavelengths, mayinclude relatively shorter wavelengths, such as visible light, amongothers. The second wavelength, or range of wavelengths, may includerelatively longer wavelengths, such as infrared (IR) and/ormillimeter-wave (MMV) wavelengths, among others. The imaged light, atthe first and/or second wavelengths, or ranges of wavelengths,optionally may include reflected or scattered illumination light,generated by the imaging system or an associated system for the purposeof enhancing images. For example, illumination light may be used inimaging radar applications, among others.

The correction of images distorted by a medium, such as the Earth'satmosphere, and/or by various optical components of an imaging system,generally can be accomplished using the mathematical principle ofdeconvolution and/or phase diversity. This principle stems from thenotion that for an arbitrary three-dimensional object, an opticalimaging system yields an image intensity distribution i(x,y,z) that isthe convolution of the object intensity distribution o(x,y,z) with thepoint spread function (PSF) s(x,y,z) describing blurring of atheoretical point source of light:

$\begin{matrix}{{{i( {x,y,z} )} = {{\int_{- \infty}^{\infty}{{x^{\prime}}{\int_{- \infty}^{\infty}{{y^{\prime}}{\int_{- \infty}^{\infty}{{z^{\prime}}{o( {x^{\prime},y^{\prime},z^{\prime}} )}{s( {{x - x^{\prime}},{y - y^{\prime}},{z - z^{\prime}}} )}}}}}}} \equiv {o( {x,y,z} ){s( {x,y,z} )}\mspace{14mu} ( {3\text{-}D\mspace{14mu} {case}} )}}},} & (1)\end{matrix}$

where {circle around (×)} is called the convolution operator. The PSFdescribes how light is spread out or blurred by the medium and/orimaging system due to diffraction and other effects as the light travelsbetween the object and image. The same relationship applies fortwo-dimensional (i.e., planar) and one-dimensional (i.e., linear)objects, but the convolution equation takes simpler forms:

$\begin{matrix}\begin{matrix}{{i( {x,y} )} = {\int_{- \infty}^{\infty}{{x^{\prime}}{\int_{- \infty}^{\infty}{{y^{\prime}}{o( {x^{\prime},y^{\prime}} )}{s( {{x - x^{\prime}},{y - y^{\prime}}} )}}}}}} \\{{\equiv {{o( {x,y} )}{s( {x,y} )}\mspace{14mu} ( {2\text{-}D\mspace{14mu} {case}} )}};}\end{matrix} & (2) \\{{i(x)} = {{\int_{- \infty}^{\infty}{{x^{\prime}}{o( x^{\prime} )}{s( {x - x^{\prime}} )}}} \equiv {{o(x)}{s(x)}\mspace{14mu} {( {1\text{-}D\mspace{14mu} {case}} ).}}}} & (3)\end{matrix}$

For simplicity, in this disclosure, the spatial dependence hereinaftertypically will be omitted from equations; e.g., the convolutionoperation will be written

i=o

s,  (4)

without regard to the number of spatial dimensions.

The goal of deconvolution is to extract an object intensity distributionfunction o, describing the actual distribution of intensity in anobject, from the measured image intensity distribution function i, whichmay be degraded by environmental and/or instrumental factors, asdescribed above. The convolution theorem of mathematics holds that theFourier transform of the convolution of two functions is the ordinaryproduct of the Fourier transforms of the functions, i.e., that

Γ(f

g)=Γ(f)Γ(g),  (5)

where Γ is the Fourier transform operator, defined in one dimension(with suitable generalizations to greater numbers of dimensions) by

$\begin{matrix}{{\Gamma ( {f(x)} )} = {{F(\omega)} = {\frac{1}{\sqrt{2\pi}}{\int_{- \infty}^{\infty}{{f(x)}^{{- }\; \omega \; x}{{x}.}}}}}} & (6)\end{matrix}$

As a result of this mathematical simplification, deconvolutiontechniques often are performed in Fourier (or frequency) space. TheFourier transform of the PSF, i.e.,

Γ(s(x))≡S(ω),  (7)

is sometimes referred to as the optical transfer function (OTF).

The following sections further describe these and other aspects of thepresent teachings, including, among others, (I) image corrections withknown point spread functions, (II) image corrections with unknown pointspread functions, (III) image corrections across wavelength regimes, and(IV) examples.

I. IMAGE CORRECTIONS WITH KNOWN POINT SPREAD FUNCTIONS

In some instances, the PSF may be a known or independently determinablefunction. This may be the case, for example, when there is a temporallyconstant medium (so that the PSF can be determined in advance), or whena point-like test object is located or can be placed near the actualobject of interest, so that the PSF may be determined from themeasurable aberration of the test object. This also may be the case insituations in which it is desirable to perform an approximate or “quickand dirty” deconvolution, for example, based on a calculated orbest-guess PSF. In any of these cases, the Fourier transform of Eq. (4)yields

$\begin{matrix}{{{\Gamma (o)} = \frac{\Gamma (i)}{\Gamma (s)}},} & (8)\end{matrix}$

and taking the inverse Fourier transform,

$\begin{matrix}{{o = {\Gamma^{- 1}( \frac{\Gamma (i)}{\Gamma (s)} )}},} & (9)\end{matrix}$

where the inverse transform is defined by

$\begin{matrix}{{\Gamma^{- 1}(F)} = {{f(x)} = {\frac{1}{\sqrt{2\pi}}{\int_{- \infty}^{\infty}{{F(\omega)}^{{\omega}\; x}{{\omega}.}}}}}} & (10)\end{matrix}$

Determining the object intensity distribution function using Eq. (9) issometimes called direct inverse filtering, and may be a suitabletechnique when the PSF is known and in the absence of significant noise.

However, even when the PSF is known, a complication may arise when—inaddition to optical aberrations arising from the medium and thecomponents of the optical system-system noise is detected as part of theimage intensity distribution. In this case, any of Eqs. (1)-(4) may bewritten

i=o

S+n,  (11)

Here, n represents the system noise. If the noise is significant, it maybe desirable for a chosen method of deconvolution to account for thenoise, as well as the PSF, when determining the object intensitydistribution. Using the convolution theorem, the Fourier transform ofEq. (11) is

Γ(i)=Γ(o)Γ(s)+Γ(n),  (12)

which can be rewritten as

I(ω)=O(ω)S(ω)+N(ω),  (13)

where I(ω), O(ω), S(ω), and N(ω) are the Fourier transforms of the imageintensity distribution function, the object intensity distributionfunction, the PSF, and the noise, respectively. In this case, the objectintensity distribution function may not be accurately recoverable simplyby taking the inverse transform of Eq. (12) or (13), and it may bedesirable to use other techniques, some of which are described below.

I.A. Wiener Filtering

One technique that may be used in the presence of significant noise,generally known as Wiener filtering, applies a linear, noise-dependentattenuating filter Φ(ω) before inverse Fourier transforming to find anestimate for the object intensity distribution function:

$\begin{matrix}{\overset{\sim}{o} \approx {{\Gamma^{- 1}( \frac{{\Phi (\omega)}{I(\omega)}}{S(\omega)} )}.}} & (14)\end{matrix}$

A goal of this technique is to find the optimal filter Φ(ω) leading tothe best estimate of the object intensity distribution function õ in Eq.(14). One such attenuating filter may be found, for example, by assumingthat the object and noise functions (i.e., o and n) are uncorrelated,and then mathematically minimizing the quadratic error between theobject estimate õ and the true object o. This minimization leads to aWiener filter of the form

$\begin{matrix}{{\Phi (\omega)} = {\frac{{{I(\omega)}^{2}} - {{N(\omega)}^{2}}}{{I(\omega)}^{2}}.}} & (15)\end{matrix}$

I.B. Richardson-Lucy Algorithm

Another technique that may be used in the presence of noise, commonlyknown as the Richardson-Lucy (RL) algorithm, maximizes the likelihoodfunction of the object intensity distribution assuming a Poissondistribution for the noise (as is often the case for photon noise). Thislikelihood function essentially is the probability of measuring the meanintensity of a large number of measurements with a single measurement.The result of maximizing the likelihood function is an iterativealgorithm, in which each successive estimate of the object intensityfunction is computed from the previous estimate until a desired degreeof convergence is reached:

$\begin{matrix}{{o^{({n + 1})} = {o^{(n)}( \frac{s \times i}{so^{(n)}} )}},} & (16)\end{matrix}$

Here, the “x” operator in the numerator on the right-hand siderepresents ordinary multiplication. The initial object function, o⁽⁰⁾,typically is the uncorrected image intensity function of a constant meanvalue.

II. IMAGE CORRECTIONS WITH UNKNOWN POINT SPREAD FUNCTIONS

When the PSF is unknown and cannot be measured directly, it may benecessary to find both the PSF and the corrected image from theavailable image data. Techniques for accomplishing this generally aretermed methods of “blind deconvolution.” In general, blind deconvolutionmethods use known or estimated information, such as physical constraintson the object intensity distribution function, the noise function,and/or the PSF effectively to reduce the number of unknowns so that thesystem is soluble.

II.A. Richardson-Lucy Extension Method

One method of blind deconvolution is an extension of the Richard-Lucy(RL) algorithm (described previously) to the case where the PSF isunknown. This method can be used alternatively to estimate the objectintensity distribution o and the PSF s. In this approach, the iterativeequation for updating the object intensity function is the same as inthe RL algorithm, i.e., as given in Eq. (16) above, and the iterativeequation for updating the PSF is given by

s ^((i+1)) =s ^((i)){(f ^((i−1)) ×g)/(s ^((i))

f^((i)))}  (17)

More details on this method of blind deconvolution can be found, forexample, in G. R. Ayers and J. C. Dainty, “Iterative Blind DeconvolutionMethod and its Applications,” Optics Letters 13 (7), 547-549 (July1988), which is incorporated herein by reference in its entirety for allpurposes.

II.B. Phase Diversity Method

Another method of blind deconvolution is based on the phase diversity oftwo measured images of the same object. In this technique, one imagecontains only the unknown aberrations, and another image of the sameobject is intentionally blurred by an additional known amount. The OTFsof the two images then will have the same magnitude, but differentphases:

S ₁(ω)=|S ₁(ω)|exp|i

(ω)|; S ₂(ω)=|S ₂(ω)|exp|i

(ω)+ρ(ω)|,  (18)

where ρ(ω) is the phase difference introduced by the intentionaldefocus, also known as the phase diversity between the two OTFs. Theobject intensity distribution function may be found by assuming aparticular type of noise and then maximizing the likelihood of thedistribution function, as will be described below in more detail, in thecontext of Gaussian and Poisson noise. Additional information about thephase diversity method of blind deconvolution can be found in R. G.Paxman et al., “Optical Misalignment Sensing and Image ReconstructionUsing Phase Diversity,” J. Opt. Soc. Am. A 5 (6), 914-923 (June 1988),which is incorporated herein by reference in its entirety for allpurposes.

II.B.1. Gaussian Noise Assumptions

Maximizing the likelihood function of the object intensity distributionfunction under Gaussian noise assumptions results in a closed-formexpression for the object intensity distribution, such that

$\begin{matrix}{{{O(\omega)} = \frac{{{I_{1}(\omega)}{S_{1}^{\star}(\omega)}} + {{I_{2}(\omega)}{S_{2}^{\star}(\omega)}}}{{{S_{1}(\omega)}}^{2} + {{S_{2}(\omega)}}^{2}}},} & (19)\end{matrix}$

where the subscripts on I and S refer to the two different diversityimages, and the symbol “*” means complex conjugate. Substituting thissolution back into the likelihood equation results in an objectivefunction in which the only unknowns are the PSF aberration parameters.Non-linear optimization techniques, such as gradient search-basedalgorithms, then can be used to find the PSF. Once the PSF is known, thedeblurred image can be recovered using standard techniques, such as theWiener filtering technique described above. More details on this methodof phase diversity blind deconvolution can be found in U.S. Pat. No.4,309,602 to Gonsalves et al., which is incorporated herein by referencein its entirety for all purposes.

II.B.2. Poisson Noise Assumptions

Another method of phase diversity blind deconvolution uses anexpectation maximization algorithm to jointly recover o and s underPoisson noise assumptions. Like the RL algorithm, this method isparticularly suitable when the dominant noise component is photon noise.This method iteratively updates the estimated restored image (i.e., theobject intensity distribution function) and the estimated PSF, so as toincrease the likelihood function at every update. The object intensitydistribution function update equation is:

$\begin{matrix}{o^{({n + 1})} = {o^{(n)}{\frac{{( {s_{1} \times i_{1}} )( {s_{1}o^{(n)}} )} + {( {s_{2} \times i_{2}} )( {s_{2}o^{(n)}} )}}{{S_{1}(0)} + {S_{2}(0)}}.}}} & (20)\end{matrix}$

The PSF update equations are found by substituting the current value ofthe image into the likelihood, and then maximizing with respect to thoseparameters. The estimates of the object intensity distribution functionand the PSFs are updated iteratively, until the change in likelihoodfrom one iteration to the next reaches any suitably small threshold.More details regarding this method may be found, for example, in R. G.Paxman et al., “Joint Estimation of Object and Aberrations by UsingPhase Diversity,” J. Opt. Soc. Am. A 9 (7), 1072-1085 (July 1982), whichis incorporated herein by reference in its entirety for all purposes.

FIG. 1 is a schematic diagram illustrating an exemplary optical system,generally indicated at 10, employing the phase diversity method of blinddeconvolution. In this diagram, an unknown object 12 transmits, emits,and/or reflects light, two representative rays of which are indicated at14, 16. These rays pass through a region of unknown turbulence 18, andthen through a converging lens 20 of the optical system. Turbulentregion 18 may represent, for example, a region of the Earth'satmosphere. More generally, turbulent region 18 may represent any othermedium and/or influence having an unknown effect on light from anobject. Although only one lens 20 is shown in FIG. 1, the optical systemgenerally may include a plurality of suitable optical components, suchas lenses and mirrors, among others.

Lens 20 refracts rays 14, 16, which then reach a beam splitter 22, whichsplits rays 14, 16 into two sets of rays 14 a, 16 a and 14 b, 16 b. Rays14 a, 16 a pass with their directions unaffected through the beamsplitter, and then converge to form an image 24 in the focal plane P oflens 20, where an image collecting device (not shown) may be positioned.Rays 14 b, 16 b, on the other hand, are reflected by the beam splitter,converge in the focal plane P′ of lens 20, and then diverge to form adiversity image 26 at a position translated a known distance D beyondplane P′, where a second image collecting device (not shown) may bepositioned. Various methods, including those described previously, amongothers, may be used to reconstruct the PSF and the corrected image fromthe two sets of image data 24, 26.

II.C. Additional Method(s)

Other methods of blind deconvolution may be suitable for determining anunknown PSF, in addition to the methods described above. These include,for example, global minimization techniques, among others, such assimulated annealing. More information regarding global minimization canbe found in B. C. McCallum, “Blind deconvolution by simulatedannealing,” Optics Communication 75(2), 101-105 (February 1990), whichis incorporated herein by reference in its entirety for all purposes.

III. IMAGE CORRECTIONS ACROSS WAVELENGTH REGIMES

The PSF is generally a wavelength dependent function; thus, applying anytechnique to correct image aberrations may involve finding the PSF inapproximately—or in some cases, exactly—the wavelength regime of thedesired image. For example, a visible-range PSF may be used to correctvisible-range images, an infrared PSF may be used to correct infraredimages, a millimeter-wave PSF may be used to correct millimeter-waveimages, and so forth. More specifically, a PSF may be determined for theprecise wavelength(s) of the image; for instance, a 630 nanometer (nm)PSF may be determined and used to correct a monochromatic 630 nm image.Precise matching of the PSF to the image in this manner is most feasiblewhen the image is either relatively monochromatic, so that only a singlePSF need be determined to correct the image accurately, or has adiscrete spectrum, so that a well-defined set of PSFs may be determinedand used to correct the image.

In some situations, an imaging system and an object of interest may beat fixed locations with respect to each other, so that the object isavailable for imaging—and image correction—for a relatively long time,and with a relatively constant medium interposed between the object andthe imaging system. In such cases, it may be possible to determine inadvance a set, or “library,” of PSFs, for various wavelength regimes,which then may be used to correct an image of the object of interest.Even if such a predetermined set of PSFs is not known, time delaysattributable to determining a wavelength-specific set of PSFs andcorrecting an image may be relatively unimportant in these cases.

However, in other situations, such as imaging from ground vehicles,airplanes, or satellites for surveillance or other purposes, the imagingsystem and the object(s) it seeks to image typically may be in a stateof relative motion, so that the nature and degree of image aberration isa (rapidly) changing function of time. Furthermore, in such cases,images may be collected continuously and “on the fly,” and it may bedesirable to correct aberrations in the images relatively quickly, sothat the corrected images may be viewed essentially in real time, i.e.,with a relatively insignificant delay between collecting the images andviewing the corresponding corrected images.

One method of accomplishing real time image correction of either staticor time-varying aberrations is simply to collect image data in thewavelength regime (or bandwidth) of interest, use the collected data todetermine the PSF for that bandwidth, and then relatively quickly applythe PSF to correct the image. This technique may be especially suitablefor correcting images in the visible regime, due to the relative easeand low expense of collecting visible image data with image collectingdevices such as, for example, infrared focal planes, CMOS image, andcharge-coupled devices (CCDs). However, other methods may be moresuitable for correcting images in other wavelength regimes, as describedbelow.

Another method of accomplishing real time image correction is to collectimage data in one wavelength regime, use the collected image data todetermine corrective information, which also may be termed an imagecorrection factor, for that regime, modify the image correction factorin a suitable manner, and then apply the modified image correctionfactor in a different wavelength regime. Suitable image correctionfactors may include, for example, a PSF determined at a particularwavelength. For instance, visible image data may be collected and usedto determine a visible regime PSF, which then may be modified andapplied to other (e.g., longer) wavelength regimes. In particular, ifthe wavefront of the sensing is of a shorter wavelength, the temporaland spatial resolution of the wavefront error will be of higher spatialand higher temporal resolution than that of a longer wavelength, andthus a corrective factor based on the shorter wavelength data can beapplied to the longer wavelength data. For instance, visible phasediversity information may be collected in one wavelength regime,appropriately modified (as will be described in more detail below), andthen used to obtain a PSF or wavefront correction in another wavelengthregime after mathematical modification. Techniques using correctiveinformation from one wavelength regime to correct images in anotherregime may be especially suitable for correcting images in relativelylong wavelength regimes, such as the infrared regime and the millimeterwave regime, since it may be expensive or otherwise impractical tocollect and process corrective information in those wavelength regimessimultaneously with imagery data.

FIG. 2 is a schematic diagram showing how visible regime image data maybe collected and used to determine corrective information, such as phasediversity information, which then may be used to correct images in boththe visible regime and in other wavelength regimes. In this approach,generally indicated at 30, an optical system measures correctiveinformation in the visible regime, as indicated at 32. Then, asindicated at 34, this corrective information may be applied in thevisible regime itself, to correct visible light images, in aconventional use of the corrective information. However, as indicated at36, 38, 40, the visible regime corrective information also may besuitably converted and then used to correct near-infrared, mid-waveinfrared, or short-wave infrared imagery, respectively, illustratingwhat may be termed “spectral agility” of the corrective information.Furthermore, as indicated at 42, the visible regime correctiveinformation may be converted and applied to correct long-wave infraredimagery, illustrating what may be termed “extreme spectral agility” ofthe corrective information. Finally, as indicated at 44, the visibleregime corrective information may be converted and applied to correctmillimeter-wave imagery, illustrating what may be termed “ultimatespectral agility” of the corrective information.

Since corrective information, including the PSF, the OTF, and phasediversity information, is dependent on wavelength, converting suchinformation from one wavelength regime to correct images in anotherregime typically will involve resealing the information according towavelength. For example, in the phase diversity method, the phase ofeach wave-front varies inversely with wavelength, so that for a givenamount of translation of diversity image 26 away from focal plane P′ inFIG. 1, a different amount of phase shift, and thus a different objectintensity distribution, may be determined for each wavelength ofinterest. More specifically, if the OTFs of the phase diverse images inthe visible regime are given by Eq. (18), i.e.,

S ₁(ω)=|S ₁(ω)|exp|i

(ω)|; S ₂(ω)=|S ₂(ω)|exp|i

(ω)+ρ(ω)|,  (21)

then the corresponding expressions for the phase shifted OTFs in anyother wavelength regime may be estimated as

$\begin{matrix}{{{{S_{1}^{\prime}(\omega)} = {{{S_{1}(\omega)}}\exp {{\; (\omega)( \frac{\lambda_{vis}}{\lambda_{new}} )}}}};}{{{S_{2}^{\prime}(\omega)} = {{{S_{2}(\omega)}}\exp {{( {{\; (\omega)} + {\rho (\omega)}} )( \frac{\lambda_{vis}}{\lambda_{new}} )}}}},}} & (22)\end{matrix}$

where λ_(vis) is the visible wavelength used for the measurement of thephase diverse images, and λ_(new) is the new wavelength at which animage correction is desired. These scaled expressions for the OTFs thenmay be used to determine the corrected object intensity distributionfunction, using one of the methods described previously, or any othersuitable method of phase diversity blind deconvolution.

Similarly, any OTF determined at one wavelength may be used to generatean OTF—and thus a PSF and a corrected image—at any other desiredwavelength, by resealing the phase of the OTF according to the ratio ofthe measured and targeted wavelengths. In other words, for example, anygeneral OTF expression S(ω)=|S(ω)exp|i

(ω)| obtained from measurements made at one wavelength may be used togenerate another OTF at a different wavelength through a transformationof the form

$\begin{matrix}{{{S^{\prime}(\omega)} = {{{S(\omega)}}\exp {{\; (\omega)( \frac{\lambda_{old}}{\lambda_{new}} )}}}},} & (23)\end{matrix}$

where λ_(old) and λ_(new) are the wavelengths at which the OTF ismeasured initially and estimated subsequently, respectively. Thus, anysuitable method of deconvolution, including methods of blinddeconvolution other than phase diversity, such as the RL extensionmethod described above, may be used to determine an OTF and/or a PSF inone wavelength regime, which then may be rescaled according to atransformation of the form given by Eq. (23) to determine an OTF and/ora PSF in any other regime of interest. The rescaled PSF then may be usedto correct imagery in the new regime using an suitable method, such asdirect inverse filtering, Wiener filtering, or with the RL algorithm,among others.

IV. EXAMPLES

The following examples describe selected aspects and embodiments of thepresent teachings. These aspects and embodiments are included forillustration and are not intended to limit or define the entire scope ofthe present teachings.

Example 1

This section describes an example of some of the techniques describedabove; see FIG. 3.

FIG. 3 is a schematic diagram showing an optical system, generallyindicated at 50, for using visible image data to correct an infraredimage using phase diversity methods. In this example, an incoming imagesignal 52 arrives at the optical system, and may encounter variousoptical components including lenses and mirrors, such as mirrors 54, 56,58, 60. These optical components may be used, for example, to focus,magnify, and/or redirect the incoming signal.

Signal 52 then arrives at a beamsplitter 62, which divides or splits thesignal into two beams 64, 66. Beamsplitters, such as beamsplitter 62,generally comprise optical devices configured to separateelectromagnetic radiation into different wavelength bands, for example,separating a visible light band from an infrared radiation band.Suitable beamsplitters (such as dichroic or multi-dichroicbeamsplitters) may operate by a variety of mechanisms, for example, bytransmitting one wavelength band while reflecting another wavelengthband, and/or by deflecting or diffracting one wavelength band to adifferent extent than another wavelength band. Suitable beamsplittersmay include prismatic materials, such as fused silica or quartz, and maybe coated with a metallic or dielectric layer havingwavelength-dependent transmission and reflection properties.Alternatively, or in addition, suitable beamsplitters may includediffractive materials or devices, such as an acousto-optic modulator. Inthe present example, beamsplitter 62 is configured at leastsubstantially to transmit visible light, and at least substantially toreflect infrared light. Thus, beam 66 passes through the beamsplitter,and contains primarily or exclusively visible wavelengths. Beam 66, onthe other hand, is reflected by the beamsplitter, and contains primarilyor exclusively only the infrared portion of the signal, which isredirected towards an infrared camera 68.

After passing through beamsplitter 62, beam 64 arrives at a visiblebeamsplitter 70, which splits beam 64 into two parts, i.e., beams 72,74, each of which contains a portion of the original visible signal andwhich may be used as phase diverse beams for applying the phasediversity method of blind deconvolution. In particular, beam 74 isredirected by beamsplitter 70 and arrives at a first visible camera 78located such that beam 74 produces a focused image in camera 78. Beam72, however, passes through beamsplitter 70 towards a second visiblecamera 76, which is configured such that beam 72 produces anout-of-focus image in camera 76 with a known amount of defocus. The twovisible images and the phase diversity information then are transmittedto a data recorder/processor 80, which may use this information tocorrect both the visible imagery detected by camera 78, and also theinfrared imagery detected by camera 68. Data recorder/processor 80 maybe a single integrated device, such as an integrated circuit board,including both processing and memory capabilities, or it may include aseparate but communicating data processor and data recorder, as is mostappropriate for a given application.

More specifically, recorder/processor 80 may be programmed with one ormore algorithms for determining phase diversity image corrections in thevisible and/or the infrared regimes from the phase diverse visibleimagery collected by cameras 76 and 78, and these corrections then maybe used to correct the imagery in those regimes. The visible regimecorrections may be determined directly from the visible phase diverseimagery, and the infrared regime corrections may be determined afterresealing the visible phase diversity information according towavelength, in the manner of Eq. (22) or (23) above. After suchwavelength resealing (if any), the corrections may be found using anysuitable phase diversity deconvolution algorithm, such as, for example,those using Gaussian or Poisson noise assumptions described previously,among others.

After applying the determined corrections to one or both of the visibleand infrared imagery, recorder/processor 80 of system 50 may beconfigured to record the corrected imagery, transmit it to a remotelocation, and/or send it to one or more “heads-up” displays (not shown)for observation by a pilot or other operator. Determining thecorrections, applying them to the imagery, and recording, transmitting,or displaying the corrected imagery all may be performed essentially inreal time, so that an operator using system 50 may be able to seecorrected imagery in one or both of the visible and infrared wavelengthregimes without significant time delays. This may make system 50particularly useful for surveillance and navigation applications.

The image correction systems described in this example may incorporatemore than one distinct algorithm for determining phase diversity imagecorrections, the results of which may be compared and a preferred methodselected to optimize image resolution. In addition, the processing costof each method may vary according to each specific type of imagery andaberration, and the system may be configured automatically to select themost cost-efficient algorithm for a given situation, or to select analgorithm that balances optimal image resolution with cost-efficiency ina predetermined manner. Furthermore, although in this example visibleregime phase diverse images may be used to correct infrared imagery, thesystem also may be configured to use visible phase diverse images tocorrect imagery in other wavelength regimes, such as in themillimeter-wave regime, among others. In general, the methods employedby this system may be employed to use phase diverse imagery in anywavelength regime to correct imagery in any other regime.

Example 2

This section describes additional techniques that optionally may be usedwith, or in lieu of, techniques described elsewhere herein for improvingimage acquisition and/or quality. See, e.g., U.S. Provisional PatentApplication Ser. No. 60/696,306, filed Jul. 1, 2005, which isincorporated herein by reference in its entirety for all purposes.

Kolmogorov developed mathematical constructs for estimating atmosphericeffects. In these constructs, the resolution limits imposed by theatmosphere vary with path length (to the 8/5^(th) power), altitude (tothe −4/3 power), and sensing wavelength (to the 1/5^(th) power). Thetemporal effects from the atmosphere also vary approximately withwavelength. Thus, based on these constructs, it is possible to measurethe phase diversity and calculate a correction for the wavefront forlonger wavelengths. Tilt of the wavefront usually is the dominantwavefront error and can be corrected merely by tilting a flat mirror,for example, as described in U.S. Provisional Patent Application Ser.No. 60/696,306, filed Jul. 1, 2005.

One method to correct for this atmospheric distortion is to employ awavefront sensor to measure the spatial and temporal phase change on theincoming light, and to use a flexible mirror to remove the measureddistortions, in essence removing the atmospheric effects in real time.The wavefront sensor can be a Shack-Hartmann sensor, which is a seriesof lenslets (or subapertures) that “sample” the incoming wavefront atthe size of (or smaller than) the Fried parameter. The size of thewavefront sensor subaperture and the spacing of the actuators that areused to deform an adaptive mirror can be expected to be less than theFried coherence cell size.

The incoming light for the wavefront sensor can be generated by alaser-induced false star and/or a real star. In such cases, theisoplanatism is significant and often limits the effectiveness ofatmospheric compensation. Anisoplanatism may not be a concern for thepresent teachings as the wavefront effort is measured along the samepath length as the sensing.

Example 3

This section provides an example of an image correction system,according to aspects of the present teachings, being used in an aircraftto correct infrared, millimeter-wave, and/or visible imagery in realtime; see FIG. 4.

FIG. 4 shows a helicopter 100 equipped with an image correction system102. Image correction system 102 may be substantially similar oridentical to system 50, described above in Example 1 (and optionallyaugmented by Example 2) and depicted in FIG. 3, and may be configured tocorrect infrared, millimeter-wave, and/or visible imagery using visibleregime phase diverse imagery. The corrected infrared or millimeter-waveimagery may be recorded and/or displayed on a heads-up display,generally indicated at 104 in FIG. 4. Thus, a pilot of helicopter 100may use system 102 to view corrected infrared or millimeter-wave imageryin real time on display 104. In addition, system 102 may provide asecond heads-up display (not shown) configured to display correctedvisible imagery, which also may be corrected using the visible phasediverse imagery collected by the system. The image correction systemsprovided by the present teachings more generally may be configured foruse in any suitable type of aircraft or airborne device (among othersupports), such as fixed-wing piloted aircraft, pilotlessremote-controlled aircraft, and/or orbiting satellites, among others.Suitable support platforms, supports, and mounting devices are describedin the following patent applications, which are incorporated herein byreference in their entireties for all purposes: U.S. patent applicationSer. No. 10/956,739, filed Oct. 1, 2004; U.S. patent application Ser.No. 10/956,738, filed Oct. 1, 2004; and U.S. Provisional PatentApplication Ser. No. 60/696,306, filed Jul. 1, 2005.

Example 4

This example describes exemplary uses and applications of imagecorrection systems, in accordance with aspects of the present teachings.

The image correction systems may be used for, or applied to, anysuitable purpose(s), including navigation and/or surveillance, amongothers. These purposes may involve collecting images at two or morewavelengths. In some cases, images may be observed, processed, and/oranalyzed for just one of these wavelengths, for example, by usingvisible image information to obtain corrective data, applying thecorrective data to an infrared image, and then using only the infraredimage for subsequent analysis). In other cases, images may be observed,processed, and/or analyzed for two or more wavelengths, for example,separately or collectively (e.g., by forming a composite image).Composite images may be straight combinations of two or more otherimages. However, in some cases, one or both of the images may beprocessed prior to or during the process of combining the images. Forexample, composite images may be formed for use in firefighting,aeronautics, surveillance, and/or the like, superimposing infraredimages of hot spots, runway lights, persons, and/or the like on visibleimages. See, e.g., U.S. Pat. No. 6,232,602, issued May 15, 2001, whichis incorporated herein by reference in its entirety for all purposes.

The disclosure set forth above may encompass multiple distinctinventions with independent utility. Although each of these inventionshas been disclosed in its preferred form(s), the specific embodimentsthereof as disclosed and illustrated herein are not to be considered ina limiting sense, because numerous variations are possible. The subjectmatter of the inventions includes all novel and nonobvious combinationsand subcombinations of the various elements, features, functions, and/orproperties disclosed herein. The following claims particularly point outcertain combinations and subcombinations regarded as novel andnonobvious. Inventions embodied in other combinations andsubcombinations of features, functions, elements, and/or properties maybe claimed in applications claiming priority from this or a relatedapplication. Such claims, whether directed to a different invention orto the same invention, and whether broader, narrower, equal, ordifferent in scope to the original claims, also are regarded as includedwithin the subject matter of the inventions of the present disclosure.

1. A method of image correction, comprising: collecting a first set ofimage data in a first wavelength regime, along with phase diversitydata; collecting a second set of image data in a second wavelengthregime, without phase diversity data; processing the first set of imagedata to determine an image correction factor for correcting the secondset of image data; and correcting the second set of image data byapplying the image correction factor to the second set of image data toobtain a corrected set of image data.
 2. The method of claim 1, whereinthe step of correcting the second set of image data is performed atleast substantially in real time.
 3. The method of 1, wherein the firstwavelength regime is any regime of shorter wavelength than the secondwavelength regime.
 4. The method of claim 1, wherein the firstwavelength regime is the visible regime.
 5. The method of claim 1,wherein the step of processing the first set of image data includesprocessing two phase diverse sets of image data using the phasediversity method of blind deconvolution.
 6. The method of claim 1,wherein the step of processing the first set of image data includesusing the Richardson-Lucy extension method of blind deconvolution. 7.The method of claim 1, wherein the first wavelength regime ischaracterized by a first wavelength, and the second wavelength regime ischaracterized by a second wavelength, and wherein the step of processingthe first set of image data includes resealing a phase portion of thefirst set of image data by a function involving the first wavelength andthe second wavelength.
 8. The method of claim 1, wherein the step ofprocessing the first set of image data includes using at least twomethods of blind deconvolution to determine the image correction factor,and further comprising selecting one of the at least two methods toimprove at least one of (A) the resolution of the corrected set of imagedata, and (B) the processing costs associated with obtaining thecorrected set of image data.
 9. The method of claim 1, furthercomprising correcting the first set of image data by applying the imagecorrection factor to the first set of image data to obtain a correctedfirst set of image data.
 10. The method of claim 9, further comprisingforming a composite image using the corrected second set of image dataand the corrected first set of image data.
 11. An image correctionsystem, comprising: a first image-collecting device for collecting afirst set of image data in a first wavelength regime; a secondimage-collecting device for collecting a second set of image data in asecond wavelength regime; and a data processor configured to process thefirst set of image data to determine an image correction factor for thesecond set of image data, and further configured to obtain a correctedset of image data by applying the image correction factor to the secondset of image data.
 12. The image correction system of claim 11, whereinthe first wavelength regime is any regime of shorter wavelength than thesecond wavelength regime.
 13. The image correction system of claim 11,wherein the first wavelength regime is the visible regime.
 14. The imagecorrection system of claim 11, wherein the data processor is configuredto process the first set of image data using a method of blinddeconvolution.
 15. The image correction system of claim 14, wherein theblind deconvolution method is the Richardson-Lucy extension method ofblind deconvolution.
 16. The image correction system of claim 11,wherein the data processor is configured to obtain the corrected set ofimage data at least substantially in real time.
 17. The image correctionsystem of claim 11, further comprising an aircraft, wherein the firstimage collecting device, the second image collecting device, and thedata processor are mounted in the aircraft.
 18. The image correctionsystem of claim 17, further comprising a heads-up display, mounted inthe aircraft, and configured to display the corrected set of image datato an operator of the aircraft.
 19. The image collection system of claim11, wherein the first set of image data arrives at the firstimage-collecting device after passing through a medium selected from theset comprising water, the atmosphere, and a solids.
 20. The imagecollection system of claim 11, wherein the data processor is configuredto correct wavefront errors caused by optical components of the system.